Dense-CaptionNet : a sentence generation architecture for fine-grained description of image semantics

Automatic image captioning, a highly challenging research problem, aims to understand and describe the contents of the complex scene in human understandable natural language. The majority of the recent solutions are based on holistic approaches where the scene is described as a whole, potentially losing the important semantic relationship of objects in the scene. We propose Dense-CaptionNet, a region-based deep architecture for fine-grained description of image semantics, which localizes and describes each object/region in the image separately and generates a more detailed description of the scene. The proposed network contains three components which work together to generate a fine-grained description of image semantics. Region descriptions and object relationships are generated by the first module, whereas the second one generates the attributes of objects present in the scene. The textual descriptions obtained as an output of the two modules are concatenated to feed as an input to the sentence generation module, which works on encoder-decoder formulation to generate a grammatically correct but single line, fine-grained description of the whole scene. The proposed Dense-CaptionNet is trained and tested using Visual Genome, MSCOCO, and IAPR TC-12 datasets. The results establish a new state-of-the-art when compared with the existing top performing methodologies, e.g., Up-Down-Captioner, Show, Attend and Tell, Semstyle, and Neural Talk, especially on complex scenes

Context-aware convolutional neural network for grading of colorectal cancer histology images

Digital histology images are amenable to the application of convolutional neural networks (CNNs) for analysis due to the sheer size of pixel data present in them. CNNs are generally used for representation learning from small image patches (e.g. 224 × 224) extracted from digital histology images due to computational and memory constraints. However, this approach does not incorporate high-resolution contextual information in histology images. We propose a novel way to incorporate a larger context by a context-aware neural network based on images with a dimension of 1792 × 1792 pixels. The proposed framework first encodes the local representation of a histology image into high dimensional features then aggregates the features by considering their spatial organization to make a final prediction. We evaluated the proposed method on two colorectal cancer datasets for the task of cancer grading. Our method outperformed the traditional patch-based approaches, problem-specific methods, and existing context-based methods. We also presented a comprehensive analysis of different variants of the proposed method

Cellular community detection for tissue phenotyping in colorectal cancer histology images

Classification of various types of tissue in cancer histology images based on the cellular compositions is an important step towards the development of computational pathology tools for systematic digital profiling of the spatial tumor microenvironment. Most existing methods for tissue phenotyping are limited to the classification of tumor and stroma and require large amount of annotated histology images which are often not available. In the current work, we pose the problem of identifying distinct tissue phenotypes as finding communities in cellular graphs or networks. First, we train a deep neural network for cell detection and classification into five distinct cellular components. Considering the detected nuclei as nodes, potential cell-cell connections are assigned using Delaunay triangulation resulting in a cell-level graph. Based on this cell graph, a feature vector capturing potential cell-cell connection of different types of cells is computed. These feature vectors are used to construct a patch-level graph based on chi-square distance. We map patch-level nodes to the geometric space by representing each node as a vector of geodesic distances from other nodes in the network and iteratively drifting the patch nodes in the direction of positive density gradients towards maximum density regions. The proposed algorithm is evaluated on a publicly available dataset and another new large-scale dataset consisting of 280K patches of seven tissue phenotypes. The estimated communities have significant biological meanings as verified by the expert pathologists. A comparison with current state-of-the-art methods reveals significant performance improvement in tissue phenotyping

Retinal vasculature segmentation by morphological curvature, reconstruction and adapted hysteresis thresholding

Automatic retinal blood vessel extraction is very important for early diagnosis and prevention of several retinal diseases. In this paper, a new retinal vasculature segmentation algorithm is proposed based on mathematical morphology, principal curvature, non-maximal suppression and hysteresis thresholding based morphological reconstruction. The blood vessels are enhanced by applying the top-hat transformation and computation of maximum principal curvature at multiple scales. Vessel centerlines are then obtained by non-maximal suppression followed by adapted hysteresis thresholding and morphological reconstruction. The principal curvature image is double thresholded and morphologically reconstructed to generate the vessel skeleton map which is the aggregate threshold for region growing of detected vessel centerlines to obtain the segmented retinal vasculature. The proposed method is evaluated using the images of two publicly available databases, the DRIVE database and the STARE database. Achieved average accuracy for DRIVE and STARE is 0.9419 and 0.9434 respectively. Experimental results show that the proposed algorithm is comparable with other approaches in accuracy, sensitivity and specificity

FABnet : feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer

Perineural invasion (PNI), lymphovascular invasion (LVI) and tumor angiogenesis have strong correlation with cancer recurrence, metastasis and poor patient survival. The accurate segmentations of nerves and microvessels can be considered as the preliminary step in objective identification of PNI, LVI and tumor angiogenic analysis in histology images. We proposed a deep network for simultaneous segmentation of microvessel and nerves in routinely used H&E-stained histology images. The network is designed as an encoder–decoder architecture with embedded feature attention blocks and an uncertainty prediction. The proposed network uses Xception residual blocks, followed by atrous spatial pyramid pooling for feature extraction at multiple scales. Feature attention blocks are used in the skip connections from encoder to decoder as well as in the decoder up-sampling, which enables the network in focusing on more salient features while making prediction for segmentation. The method is evaluated using 7780 images of size 512 × 512 pixels, extracted from 20 WSIs of oral squamous cell carcinoma tissue at 20× magnification. The ensemble of network outputs at test time is used to obtain a better segmentation result and simultaneous generation of network prediction uncertainty map. The proposed network achieves state-of-the-art results compared to currently used deep neural networks for semantic segmentation (FCN-8, U-Net, Segnet and DeepLabV3+). The proposed network also gives robust segmentation performance when applied to the full digital whole slide image

Delineation of blood vessels in pediatric retinal images using decision trees-based ensemble classification

A new retinal blood vessel segmentation algorithm was developed and tested with a shared database. The observed accuracy, speed, robustness and simplicity suggest that the algorithm may be a suitable tool for automated retinal image analysis in large population-based studies

Retinal Vasculometry Associations with Cardiometabolic Risk Factors in the European Prospective Investigation of Cancer-Norfolk Study.

PURPOSE: To examine associations between retinal vessel morphometry and cardiometabolic risk factors in older British men and women. DESIGN: Retinal imaging examination as part of the European Prospective Investigation into Cancer-Norfolk Eye Study. PARTICIPANTS: Retinal imaging and clinical assessments were carried out in 7411 participants. Retinal images were analyzed using a fully automated validated computerized system that provides novel measures of vessel morphometry. METHODS: Associations between cardiometabolic risk factors, chronic disease, and retinal markers were analyzed using multilevel linear regression, adjusted for age, gender, and within-person clustering, to provide percentage differences in tortuosity and absolute differences in width. MAIN OUTCOMES MEASURES: Retinal arteriolar and venular tortuosity and width. RESULTS: In all, 279 802 arterioles and 285 791 venules from 5947 participants (mean age, 67.6 years; standard deviation [SD], 7.6 years; 57% female) were analyzed. Increased venular tortuosity was associated with higher body mass index (BMI; 2.5%; 95% confidence interval [CI], 1.7%-3.3% per 5 kg/m2), hemoglobin A1c (HbA1c) level (2.2%; 95% CI, 1.0%-3.5% per 1%), and prevalent type 2 diabetes (6.5%; 95% CI, 2.8%-10.4%); wider venules were associated with older age (2.6 μm; 95% CI, 2.2-2.9 μm per decade), higher triglyceride levels (0.6 μm; 95% CI, 0.3-0.9 μm per 1 mmol/l), BMI (0.7 μm; 95% CI, 0.4-1.0 per 5 kg/m2), HbA1c level (0.4 μm; 95% CI, -0.1 to 0.9 per 1%), and being a current smoker (3.0 μm; 95% CI, 1.7-4.3 μm); smoking also was associated with wider arterioles (2.1 μm; 95% CI, 1.3-2.9 μm). Thinner venules were associated with high-density lipoprotein (HDL) (1.4 μm; 95% CI, 0.7-2.2 per 1 mmol/l). Arteriolar tortuosity increased with age (5.4%; 95% CI, 3.8%-7.1% per decade), higher systolic blood pressure (1.2%; 95% CI, 0.5%-1.9% per 10 mmHg), in females (3.8%; 95% CI, 1.4%-6.4%), and in those with prevalent stroke (8.3%; 95% CI, -0.6% to 18%); no association was observed with prevalent myocardial infarction. Narrower arterioles were associated with age (0.8 μm; 95% CI, 0.6-1.0 μm per decade), higher systolic blood pressure (0.5 μm; 95% CI, 0.4-0.6 μm per 10 mmHg), total cholesterol level (0.2 μm; 95% CI, 0.0-0.3 μm per 1 mmol/l), and HDL (1.2 μm; 95% CI, 0.7-1.6 μm per 1 mmol/l). CONCLUSIONS: Metabolic risk factors showed a graded association with both tortuosity and width of retinal venules, even among people without clinical diabetes, whereas atherosclerotic risk factors correlated more closely with arteriolar width, even excluding those with hypertension and cardiovascular disease. These noninvasive microvasculature measures should be evaluated further as predictors of future cardiometabolic disease.EPIC was funded by the Medical Research Council, UK (G0401527), and Research into Ageing, UK (262). The retinal vessel morphometry work was supported by the Medical Research Council Population and Systems Medicine Board (MR/L02005X/1) and British Heart Foundation (PG/15/101/31889). Prof Foster has received additional support from the Richard Desmond Charitable Trust (via Fight for Sight) and the Department for Health through the award made by the National Institute for Health Research to Moorfields Eye Hospital and the UCL Institute of Ophthalmology for a Biomedical Research Centre. The views expressed in this article are those of the authors and not necessarily those of the Department for Health